AV synchronization system

ABSTRACT

In synchronization of a conventional AV system, when a stream as the master stops, a system clock cannot be corrected for synchronization. Since the correction is consistently performed even when an error is small, overhead of correction is large. On the other hand, when an error is large, sharp correction causes a feeling that something wrong. In the present invention, when the stream as the master stops, previously registering the degree of priority regarding the master, consideration of a corrected period or a previous error for the system clock, or transmitting a stream containing information regarding the master can allow for other process to serves as the master. Accordingly, it is possible to provide a method for synchronization with maintaining correction of the system clock. When a corrected error is large, gradual correction or reference can provide a system with normal feeling. Furthermore, when an error of correction is small, omission of correction or reference can provide a system with less overhead.

RELATED APPLICATIONS

This application is a continuation of application Ser. No. 10/847,302,filed on May 18, 2004 now U.S. Pat. No. 7,176,978, which in turn claimsthe benefit of Japanese Patent Application No. 2003-355064, filed onOct. 15, 2003, the disclosures of which Applications are incorporated byreference herein.

FIELD OF THE INVENTION

The present invention relates to an AV synchronization system, whichdecompresses digital-compressed audio (sound) data, video (image) data,text data, still-image data, or the like, to reproduce them with beingsynchronized with each other.

BACKGROUND OF THE INVENTION

Recently, according to upgrading of information infrastructure and theprogress in multimedia technology, an AV synchronization system, whichreads an AV stream through various transmission lines and media, andreproduces it, is desired and becomes commercial.

Some systems uses a stable line such as a digital broadcast, and othersystems read a stream through an unstable line, where a transmissionrate varies according to the degree of line congestions, such asinternet and wireless LAN.

In addition, in a home PC (personal computer), software capable ofediting streams becomes commercial, and various stream data can becreated personally.

In a method for synchronization in a conventional AV synchronizationreproduction apparatus, when processing a plurality kinds of data suchas audio data, video data, in parallel, one process corresponding to onekind of the data serves as the master, and its output timing is used asreference timing so that the other kinds of data are provided withsynchronized with the reference timing.

Since audio stop causes a feeling that something is wrong, in order tosynchronize between audio and video, an audio processing serves as themaster, and a system clock is corrected based on timing information ofaudio taken from the stream, as disclosed in Japanese Laid-OpenedPublication KOKAI No. HEI 7-303240, for example. In video reproduction,the system clock is compared with the timing information taken from thevideo stream. In order to perform synchronization, when the video leads,video reproduction stops to adjust timing, on the other hand, when videodelays, video reproduction accelerates.

FIG. 18 is a chart showing a flow of conventional synchronizationprocessing in the case that a stream containing audio, video, text and astill-image is reproduced based on the audio master. In FIG. 18,numerals 201, 202, 203, 204, and 205 represent a clock process, an audioprocess, a video process, a text process, and a still-image process,respectively. Thick lines represent the clock process, and the audioprocess 202 as the master.

In the case of the audio master, timing information PTS (PresentationTime Stamps) is taken from a stream to reproduce, and audio is providedbased on the timing of a clock 201 subjected to correction 206. Theothers, the video process 203, the text process 204, and the stillpicture process 205 are performed based on this clock as reference 207.For example, in the case of the video process 203, its timinginformation in a stream is taken from the video stream, and the video isprovided at the same time with the referenced clock 201 to synchronize.An upward arrow directing toward the clock represents correction ofclock. A downward arrow drawn from the clock represents reference ofclock.

FIG. 19 is a chart showing operation in a conventional system in thecase that the audio data stops in the audio master. After that, since aconventional system remains the audio master, the timing information PTScannot be taken from the audio. As a result, the clock cannot becorrected, therefore, the video process, the text process, and the stillpicture process are continued being performed without being synchronizedafter the stop.

In a conventional method, since the master is consistently fixed,reference time cannot be obtained, when the audio data stops caused bytrouble of a transmission line, or when the audio data is exhausted at amidpoint during edition of a stream, in reproduction in the case of theaudio master, for example.

In addition, when an error of clock is extremely large, correctioncauses a large image skip, sound skip, or the like.

Additionally, even when an error of clock is small, since the correctionand reference are consistently performed, this causes additionaloverhead.

SUMMARY OF THE INVENTION

Therefore, it is a primary object of the present invention to provide anAV synchronization system capable of maintaining proper synchronizationbetween a plurality of streams even when the master stream stops orcompletes. It is another object of the present invention to prevent alarge image skip, sound skip, or the like, and to reduce overhead.

To achieve the above object, in one aspect of the invention, an AVsynchronization system comprises a multi-separating portion forseparating a multiplexed stream, in which a plurality of streams aremultiplexed, into each of the streams; a plurality of buffers fortemporarily storing each of the separated streams respectively; aplurality of decoding portions for decoding the stream stored in each ofthe buffers respectively; a data providing portion for providing thedecoded data from the decoding portion with synchronized with a systemclock; and a plurality of time information managing portions foracquiring time information from the decoded stream, correcting thesystem clock based upon the time information when a master correctingthe system clock is registered, and switching one time informationmanaging portion in charge of the stream process of the master intoanother time information managing portion when a stream process of themaster stops.

Accordingly, changing a process of the master can maintainsynchronization even when the stream process of the master stops.

In the present invention, the following constructions for changing aprocess corresponding to the master can be used.

1). The master registered in the time information managing portionincludes the degree of priority, and when the stream process in the timeinformation managing portion with the highest degree of priority stops,the time information managing portion is switched into another timeinformation managing portion with the next highest degree of priority.In this case, it is possible to maintain proper synchronization betweena plurality of streams with optimizing a stream corresponding to thehighest degree of priority.

2). When the stream process of the master in the time informationmanaging portion stops, the time information managing portion isswitched into another time information managing portion acquiring thetime information from the stream at the earliest time. In this case, itis possible to correct a clock at the earliest time irrelevant of changeof the master.

3). When the stream process of the master in the time informationmanaging portion stops, the time information managing portion isswitched into another time information managing portion acquiring thetime information with the smallest period from the stream. In this case,it is possible to correct a clock at the earliest time irrelevant ofchange of the master.

4). When the stream process of the master in the time informationmanaging portion stops, the time information managing portion isswitched into another time information managing portion with the periodcorrected for the system clock closest to the time information managingportion. In this case, when the plurality of streams includes two kindsof audio streams, for example, it is possible to change other kinds ofaudio stream.

5). When the stream process of the master in the time informationmanaging portion stops, the time information managing portion isswitched into another time information managing portion with thesmallest error to a previous system clock. In this case, a streamprocess with a small error serves as the master to maintainsynchronization.

6). The multiplexed stream includes information of the degree ofpriority regarding a candidate of master in the stream process, when thestream process of the master in the time information managing portionwith the highest degree of priority regarding a candidate of masterstops, the time information managing portion is switched into anothertime information managing portion with the next highest degree ofpriority regarding a candidate of master. In this case, setting thedegree of priority regarding a candidate of master can provideflexibility to arbitrarily select a process, which serves as the master.

In addition, an AV synchronization system according to the presentinvention comprises a multi-separating portion for separating amultiplexed stream, in which a plurality of streams with instructioninformation containing whether the master correcting the system clock isor not are multiplexed, into each of the streams; a plurality of buffersfor temporarily storing each of the separated streams respectively; aplurality of decoding portions for decoding the stream stored in each ofthe buffers respectively; a data providing portion for providing thedecoded data from the decoding portion with synchronized with a systemclock; and a plurality of time information managing portions acquiringtime information from the decoded stream, wherein one of the timeinformation managing portion instructed as the master by the instructioninformation, serves as the master to correct the system clock, and whenthe time information managing portion is not instructed as the master,the time information managing portion is switched into another timeinformation managing portion.

Accordingly, it is possible to change a process, which serves as themaster, in any time when necessary, irrelevant of whether stop of astream process of the master occurs or not. Therefore, optimizedsynchronization can be obtained.

Additionally, an AV synchronization system according to the presentinvention comprises a multi-separating portion for separating amultiplexed stream, in which a plurality of streams are multiplexed,into each of the streams; a plurality of buffers for temporarily storingeach of the separated streams respectively; a plurality of decodingportions for decoding the stream stored in each of the buffersrespectively; a data providing portion for providing the decoded datafrom the decoding portion with synchronized with a system clock; and aplurality of time information managing portions acquiring timeinformation from the decoded stream, wherein one of the time informationmanaging portions registered as the master, serves as the master tocorrects the system clock based on the time information, and when thestream is in a predetermined state registered previously such assilence, the time information managing portion is switched into anothertime information managing portion.

According to this construction, condition for changing the master canarbitrarily be specified, therefore, this can provide flexibility.

Moreover, an AV synchronization system according to the presentinvention comprises a multi-separating portion for separating amultiplexed stream, in which a plurality of streams are multiplexed,into each of the streams; a plurality of buffers for temporarily storingeach of the separated streams respectively; a plurality of decodingportions for decoding the stream stored in each of the buffersrespectively; a data providing portion for providing the decoded datafrom the decoding portion with synchronized with a system clock; and aplurality of time information managing portions acquiring timeinformation from the decoded stream, wherein one of the time informationmanaging portions registered as the master, serves as the master tocorrects the system clock based on the time information, and whenreproduction speed varies such that the state changes from normalreproduction to fast forward reproduction, or rewind reproduction, thetime information managing portion is switched into another timeinformation managing portion.

According to this construction, it is possible to provide auser-friendly system with normal feeling, primarily based on video. Thereason is that sound can be produced at the rate with being adjusted toreproduction condition of video.

In addition, it is preferable that, when the corrected error to thesystem clock is greater than a predetermined value, the time informationmanaging portion corrects so that the corrected value becomes smaller.In this case, when the corrected error to the system clock is greaterthan a predetermined value, gradual correction can reduce negativeinfluence on other devices caused by the correction.

Additionally, it is preferable that, when the error between the timeinformation and the time information obtained by to another timeinformation managing portion is greater than a predetermined value, thetime information managing portion corrects so that the error graduallyreduces. In this case, it is possible to reduce occurrence of problem,such as an image skip, caused by large correction.

It is preferable that, after the time information managing portion isswitched into another time information managing portion to serve as themaster as a result of the stop of the stream process of the master, whenthe stream recovers, the former time information managing portion servesas the master again. In this case, since the timing information acquiredby the one of the timing information managing portions serves as themaster again, suitable condition can be obtained.

It is preferable that, when the stream process of the master stops, thetime information managing portion calculates a corrected value for thesystem clock based on a previous corrected value and continuesperforming correction for a while. In this case, when the streamrecovers within a predetermined time, the process corresponding to themaster can remain as the master. Thus, when the stream recoversrelatively early, the process corresponding to the master can remain asthe master. Accordingly, since correction of clock is performed based onan original process subsequently, this can improve stability ofoperation.

In addition, it is preferable that, when the stream process of themaster stops, in consideration of a previous error for the system clock,when an error is grater than a reference value, the time informationmanaging portion is switched into another time information managingportion to serve as the master, while when the error is less than thereference value, the time information managing portion is not switchedto continue the stream processing for a while. In th is case, when aprocess as a master recovers within predetermined time, the processremains as a master. Even if the stream process corresponding to themaster stops, when the stream recovers relatively early, the processcorresponding to the master can remain as the master. Accordingly, sincean original process continues subsequently, this can improve stabilityof operation.

It is preferable that, in consideration of an average error of previouscorrected values for the system clock of the master, when an error issmall, the time information managing portion omits a correction processso as to reduce overhead. In this case, reducing number of correctioncan reduce overhead in correction.

It is preferable that, when a previous average error of the system clockis small in a stream process other than the master, the time informationmanaging portion omits a reference process so as to reduce overhead. Inthis case, reducing number of reference can reduce overhead inreference.

In another aspect of the invention, an AV synchronization systemcomprises a multi-separating portion for separating a multiplexedstream, in which a plurality of streams are multiplexed, into each ofthe streams; a plurality of buffers for temporarily storing each of theseparated streams respectively; a plurality of decoding portions fordecoding the stream stored in each of the buffers respectively; a dataproviding portion for providing the decoded data from the decodingportion with synchronized with a system clock; and a time informationmanaging portion for regarding one of the plurality of streams as amaster, acquiring time information from the decoded stream regarded as amaster, and correcting the system clock based upon the time information;wherein the time information managing portion regards the other one ofthe plurality of streams as a new master when a stream process of astream regarded as an original master is stops, acquires new timeinformation from the decoded stream regarded as the new master, andcorrects the system clock based upon the new time information.

In the present invention, the following constructions for regardingother one of the plurality of streams as a new master can be used.

1).The time information managing portion manages information of thedegree of priority regarding a candidate of master for each stream, andregards the other one of the plurality of streams, which has the highestdegree of priority except for the stream regarded as the originalmaster, as the new master when the stream process of the stream regardedas the original master stops.2). When the stream process of the stream regarded as the originalmaster stops, the time information managing portion regards the otherone of the plurality of streams, which is able to be acquired timeinformation at the earliest time except for the stream regarded as themaster, as a new master.3).When the stream process of the stream regarded as the master stops,the time information managing portion regards the other one of theplurality of streams, which is able to be acquired time information withthe smallest period except for the stream regarded as the originalmaster, as a new master.4). When the stream process of the stream regarded as the originalmaster stops, the time information managing portion regards the otherone of the plurality of streams, which is able to be acquired timeinformation that is able to correct the system clock with the closestperiod to the time information of the stream regarded as the originalmaster, as a new master.5). When the stream process of the stream regarded as the originalmaster stops, the time information managing portion regards the otherone of the plurality of streams, which is able to the acquired timeinformation whose error to a previous system clock is smallest exceptfor the stream regarded as the original master, as a new master.6).The multiplexed stream includes information of the degree of priorityregarding a candidate of master, and when the stream process of thestream regarded as the original master stops, the time informationmanaging portion regards the other one of the plurality of streams,whose the degree of priority is highest except for the stream regardedas the original master, as a new master.

In addition, it is preferable that, after the time information managingportion regards the other one stream as the new master as a result ofthe stop of the stream process of the stream regarded as the originalmaster, when the stream process of the stream regarded as the originalmaster recovers, the time information managing portion regards thestream regarded as the original master as the master again.

It is preferable that, when the stream process of the stream regarded asthe original master stops, the time information managing portioncalculates a corrected value for the system clock based on a previouscorrected value and continues performing correction for a predeterminedperiod based on the calculated corrected value.

It is preferable that, when the stream process of the stream regarded asoriginal the master stops, the time information managing portion regardsthe other one stream as the new master on condition that a previouserror for system clock of the master is grater than a reference value,while calculates a corrected value for the system clock based on aprevious corrected value and continues performing correction based onthe calculated corrected value for a predetermined period on conditionthat a previous error for system clock of the master is less than areference value.

In addition, an AV synchronization system comprises a multi-separatingportion for separating a multiplexed stream, in which a plurality ofstreams are multiplexed, into each of the streams; a plurality ofbuffers for temporarily storing each of the separated streamsrespectively; a plurality of decoding portions for decoding the streamstored in each of the buffers respectively; a data providing portion forproviding the decoded data from the decoding portion with synchronizedwith a system clock; and a time information managing portion forregarding one of the plurality of streams as a master, acquiring timeinformation from the decoded stream regarded as a master, and correctingthe system clock based upon the time information; wherein the pluralityof streams include instruction information containing whether the eachof streams itself is or not the master, which varies currently, intoeach of the streams, and the time information managing portion regards astream whose the instruction information indicates that the streamitself is the master currently as the master.

Additionally, an AV synchronization system comprises a multi-separatingportion for separating a multiplexed stream, in which a plurality ofstreams are multiplexed, into each of the streams; a plurality ofbuffers for temporarily storing each of the separated streamsrespectively; a plurality of decoding portions for decoding the streamstored in each of the buffers respectively; a data providing portion forproviding the decoded data from the decoding portion with synchronizedwith a system clock; and a time information managing portion forregarding one of the plurality of streams as a master, acquiring timeinformation from the decoded stream regarded as a master, and correctingthe system clock based upon the time information; wherein when thestream regarded as an original master is in a predetermined state, thetime information managing portion regards the other one of the pluralityof streams as a new master.

In this AV synchronization system, the stream regarded as the originalmaster can be audio stream and the predetermined state can be silentstate.

Additionally, an AV synchronization system comprises: a multi-separatingportion for separating a multiplexed stream, in which a plurality ofstreams are multiplexed, into each of the streams; a plurality ofbuffers for temporarily storing each of the separated streamsrespectively; a plurality of decoding portions for decoding the streamstored in each of the buffers respectively; a data providing portion forproviding the decoded data from the decoding portion with synchronizedwith a system clock; and a time information managing portion forregarding one of the plurality of streams as a master, acquiring timeinformation from the decoded stream regarded as a master, and correctingthe system clock based upon the time information; wherein when thereproduction speed of the multiplexed stream is other than normalreproduction speed, the time information managing portion regards theother one of the plurality of streams as a new master.

Additionally, an AV synchronization system comprises a multi-separatingportion for separating a multiplexed stream, in which a plurality ofstreams are multiplexed, into each of the streams; a plurality ofbuffers for temporarily storing each of the separated streamsrespectively; a plurality of decoding portions for decoding the streamstored in each of the buffers respectively; a data providing portion forproviding the decoded data from the decoding portion with synchronizedwith a system clock; and a time information managing portion forregarding one of the plurality of streams as a master, acquiring timeinformation from the decoded stream regarded as a master, and correctingthe system clock based upon the time information; wherein when acorrected error value for the system clock based on the time informationis greater than a predetermined value, the time information managingportion corrects the system clock so that the error between the timeinformation of the stream regarded as the master and the system clockgradually reduces.

Additionally, an AV synchronization system comprises a multi-separatingportion for separating a multiplexed stream, in which a plurality ofstreams are multiplexed, into each of the streams; a plurality ofbuffers for temporarily storing each of the separated streamsrespectively; a plurality of decoding portions for decoding the streamstored in each of the buffers respectively; a data providing portion forproviding the decoded data from the decoding portion with synchronizedwith a system clock; and a time information managing portion forregarding one of the plurality of streams as a master, acquiring timeinformation from the decoded stream regarded as a master, correcting thesystem clock based upon the time information, and correcting an outputof a stream other than the stream regarded as the master based upon thesystem clock; wherein when a corrected error value for the output of thestream other than the stream regarded as the master based on the systemclock is greater than a predetermined value, the time informationmanaging portion corrects the output of the stream other than the streamregarded as the master so that the error between the system clock andthe output of the stream other than the stream regarded as the mastergradually reduces.

Additionally, an AV synchronization system comprises a multi-separatingportion for separating a multiplexed stream, in which a plurality ofstreams are multiplexed, into each of the streams; a plurality ofbuffers for temporarily storing each of the separated streamsrespectively; a plurality of decoding portions for decoding the streamstored in each of the buffers respectively; a data providing portion forproviding the decoded data from the decoding portion with synchronizedwith a system clock; and a time information managing portion forregarding one of the plurality of streams as a master, acquiring timeinformation from the decoded stream regarded as a master, and correctingthe system clock based upon the time information; wherein when anaverage value of previous corrected errors for the system clock based onthe time information is less than a predetermined value, the timeinformation managing portion omits the correction.

Moreover, an AV synchronization system comprises a multi-separatingportion for separating a multiplexed stream, in which a plurality ofstreams are multiplexed, into each of the streams; a plurality ofbuffers for temporarily storing each of the separated streamsrespectively; a plurality of decoding portions for decoding the streamstored in each of the buffers respectively; a data providing portion forproviding the decoded data from the decoding portion with synchronizedwith a system clock; and a time information managing portion forregarding one of the plurality of streams as a master, acquiring timeinformation from the decoded stream regarded as a master, correcting thesystem clock based upon the time information, and correcting an outputof a stream other than the stream regarded as the master based upon thesystem clock; wherein when an average value of previous corrected errorsfor the output of the stream other than the stream regarded as themaster based on the system clock is less than a predetermined value, thetime information managing portion omits the correction for the output ofthe stream other than the stream regarded as the master.

As mentioned above, according to the present invention, when the streamprocess corresponding to the master stops, the timing information, whichis acquired from other decoded stream, serves as the master, thus, it ispossible to maintain proper synchronization between a plurality ofstreams. Further, in the case that an error of clock is large, gradualcorrection can reduce a sharp image skip, sound skip, and so on.Furthermore, in the case that an error of clock is small, omission ofcorrection or reference can provide a system with less overhead.

BRIEF DESCRIPTION OF THE DRAWINGS

Other objects of the present invention will become apparent from thefollowing detailed description and the appended claims. Furthermore,other advantages of the present invention will become apparent to thoseskilled in the art from the following detailed description.

FIG. 1 is a block diagram showing a construction of an AVsynchronization system according to a concrete embodiment of the presentinvention;

FIG. 2 is a timing chart showing operation in the case that audio datastops in the audio master in a concrete embodiment of the presentinvention;

FIG. 3 is a timing chart showing a processing flow performed by amethod, in which a process capable of correction at the next earliesttime is changed as the master, when a stream process corresponding tothe master stops, in a concrete embodiment;

FIG. 4 is a timing chart showing a processing flow performed byprocessing, in which a process with the smallest period is changed asthe master, when the stream process corresponding to the master stops,in a concrete embodiment;

FIG. 5 is a timing chart showing a processing flow performed by amethod, in which a process with the period corrected for the systemclock closest to a process, which currently serves as the master, ischanged as the master, when the stream process corresponding to themaster stops, in a concrete embodiment;

FIG. 6 is a timing chart showing a processing flow performed byprocessing, in which a device with the smallest error between a systemclock and a previous system clock taken from a stream, when the streamprocess corresponding to the master stops, in a concrete embodiment;

FIG. 7 is a timing chart showing a processing flow in the case that astream, which includes information containing the degree of priority asa candidate of the master, is transmitted in a concrete embodiment ofthe present invention;

FIG. 8 is a timing chart showing a processing flow in the case that astream, which includes information regarding a process to currentlyserve as the master, is transmitted, in a concrete embodiment of thepresent invention;

FIG. 9 is a timing chart showing a processing flow performed byprocessing, in which, when a process as the master falls within apredetermined condition, other process serves as the master, in aconcrete embodiment;

FIG. 10 is a timing chart showing a processing flow performed byprocessing, in which, when reproduction speed varies caused byinstruction of fast forward reproduction, or the like, other deviceserves as the master, in a concrete embodiment of the present invention;

FIG. 11 is a timing chart showing a processing flow, in which an errorregarding a corrected value is gradually reduced when the error isgreater than a predetermined value, in the case that a clock iscorrected based on the master, in order to prevent a large image skip ofother processing and so on, caused by sharp correction, in a concreteembodiment;

FIG. 12 is a timing chart showing a processing flow, in which, when anerror to PTS is greater than a predetermined value with reference to aclock in a processes other than the master, the error is graduallyreduced, in order to prevent a large image skip of other processing andso on, caused by sharp output correction, in a concrete embodiment;

FIG. 13 is a timing chart showing a processing flow in which a processcorresponding to the original master serves as the master again when theprocess corresponding to the original master is performed again, afterother processing serves as the master in the case that the processcorresponding to the original master stops, in a concrete embodiment;

FIG. 14 is a timing chart showing a processing flow, in which correctionis continued based on a process as the master by calculating based on aprevious corrected value for a while when a process corresponding to themaster stops, and other processing serves as the master when the processcorresponding to the master does not recover, in a concrete embodiment;

FIG. 15 is a timing chart showing a processing flow, in which, when astream process corresponding to the master stops, in consideration of aprevious error of corrected value, when the previous error of correctedvalue is grater than a reference value, other process is changed as themaster, while when the error is less than the reference value, thestream process corresponding to the master remain as the master withoutcorrection for a while, in a concrete embodiment;

FIG. 16 is a timing chart showing a processing flow, in which correctionis omitted when an average error of corrected values in several times ofprevious correction, so as to reduce overhead, in the case thatcorrection is performed based on a process as the master, in a concreteembodiment;

FIG. 17 is a timing chart showing a processing flow, in which referenceis omitted when an error between reference values in the latest severaltimes and PTS of a stream is small, in the case that each device otherthan the master references a clock, in a concrete embodiment;

FIG. 18 is a chart showing synchronization processing in the case that astream containing audio, video, text and a still-image is reproducedbased on the audio master, in a conventional art;

FIG. 19 is a timing chart showing operation in the case that audio datastops in the audio master in a conventional art.

DETAILED DESCRIPTION OF THE INVENTION

The following description will describe preferred concrete embodimentsaccording to the present invention with reference to the drawings.

FIG. 1 shows a construction of an AV synchronization system according toa preferred embodiment 1 of the present invention. This AVsynchronization system handles with a stream, which audio and video aremultiplexed thereto.

In the AV synchronization system of this concrete example, a streaminput portion 101 receives a stream and a multiplex separation portion102 separates it into an audio stream and a video stream. The audiostream is supplied to an audio buffer 103 and the video stream issupplied to a video buffer 108. An audio decoding portion 104 decodesthe audio stream. An audio time information management portion 105 takestime information PTS (Presentation Time Stamps) from the decoded stream,and corrects a system clock based on its timing in the case of the audiomaster.

The processed data is sent to an audio providing portion 106, and isproduced as sound. A video decoding portion 109 decodes the videostream. A video time information management portion 110 takes timeinformation PTS from the decoded video stream, and compared with thesystem clock. A video providing portion 111 provides it in the sametiming.

FIG. 2 is a timing chart showing operation in the case that audio datastops in the audio master. Thick lines represent the clock process, andthe audio process as the master. An upward arrow directing toward theclock represents correction of clock. A downward arrow drawn from theclock represents reference of clock. A dotted line represents a stop ofaudio data. When the audio data stops, a video process serves as themaster. Accordingly, the timing information PTS can be taken from thestream in a video process. Thus, the clock can be correctedsubsequently, and it is possible to maintain synchronization in thevideo master.

In this example, the degree of priority is previously registered by afunction “Master_regist”. Audio is registered as the highest degree ofpriority, and video is registered as the second highest degree ofpriority. Accordingly, first, audio serves as the master, and videoserves as the master when the audio stream stops.

FIG. 3 is a timing chart showing a processing flow performed by amethod, in which a process capable of correction at the next earliesttime is changed as the master, when a stream process corresponding tothe master stops.

In an illustrated example, when the audio stream stops, a still-imageprocess can acquire the timing information PTS at the earliest time.Accordingly, since the still-image process serves as the master, theclock can be corrected quickly, thus, it is possible to maintainsynchronization.

FIG. 4 is a timing chart showing a processing flow performed byprocessing, in which a process with the smallest period is changed asthe master, when the stream process corresponding to the master stops.

In an illustrated example, the video process has the smallest period.Accordingly, a video process serves as the master. Subsequently, theclock can be minutely corrected based on the video process, and it ispossible to maintain synchronization.

FIG. 5 is a timing chart showing a processing flow performed by amethod, in which a process with the period corrected for the systemclock closest to a process, which currently serves as the master, ischanged as the master, when the stream process corresponding to themaster stops.

In the case of a two-channel broadcast such as a bilingual broadcastcurrently transmitted, retrieving other channel with the same period cancause audio 2 to serve as the master. Accordingly, the audio 2 serves asthe master, it is possible to reproduce without a feeling that somethingwrong.

FIG. 6 is a timing chart showing a processing flow performed byprocessing, in which a device with the smallest error between a systemclock and a previous system clock taken from a stream, when the streamprocess corresponding to the master stops. Accordingly, the clock can becorrected based on a text process, and it is possible to maintainsynchronization.

In an illustrated example, the text process has the smallest previouserror, and thus serves as the master.

FIG. 7 is a timing chart showing a processing flow in the case that astream, which includes information containing the degree of priority asa candidate of the master, is transmitted. This can allow to change thedegree of priority to perform according to the contents of a stream.

In an illustrated example, spare bits of a stream contain the priorityof master, which represents first and second as the audio processes,third as the video process, and fourth priority as the text process.Thus, the audio 2 process as second priority serves as the master, whenthe audio 1 process as the first priority stops.

The following description will describe a method for switching aprocess, which serves as the master, irrelevant to a data stop in theprocess as a mater.

FIG. 8 is a timing chart showing a processing flow in the case that astream, which includes information regarding a process to currentlyserve as the master, is transmitted.

Spare bits of a stream containing information of a current master aretransmitted. In an illustrated example, the stream including informationas audio 1 master is transmitted. Then, a process, which serves as themaster, is changed from the audio 1 process to the audio 2 process bytransmitting the steam including information as audio 2 master. Thechange is irrelevant to a stop of audio 1 master.

FIG. 9 is a timing chart showing a processing flow performed byprocessing, in which, when a process as the master falls within apredetermined condition, other process serves as the master.

In an illustrated example, silence in audio is registered as thecondition. When the audio becomes silence, a video process automaticallychanges as the master. It is preferable that all video is reproduced andthen the silent audio is adjusted rather than that all audio is producedand then the video is adjusted. The reason is that this can provide auser-friendly system with normal feeling. The change is irrelevant to astop of audio master.

FIG. 10 is a timing chart showing a processing flow performed byprocessing, in which, when reproduction speed varies caused byinstruction of fast forward reproduction, or the like, other deviceserves as the master.

Variation of reproduction speed is previously registered so that a videoprocess changes as master. This can allow for the video process to serveas the master automatically when fast forward reproduction. Thus, videocan be reproduced on a priority basis, whereby the video can beconstantly displayed when fast forward reproduction. Accordingly, it ispossible to achieve when fast forward reproduction with normal feeling.The change is irrelevant to a stop of audio master.

The following description will describe a method for correcting a clock.

FIG. 11 is a timing chart showing a processing flow, in which an errorregarding a corrected value is gradually reduced when the error isgreater than a predetermined value, in the case that a clock iscorrected based on the master, in order to prevent a large image skip ofother processing and so on, caused by sharp correction.

When a corrected error is greater than a certain value, the correctionis revised within a certain rate of the error (in this case, 70%). Thiscan reduce influence on other device caused by the correction.

FIG. 12 is a timing chart showing a processing flow, in which, when anerror to PTS is greater than a predetermined value with reference to aclock in a processes other than the master, the error is graduallyreduced, in order to prevent a large image skip of other processing andso on, caused by sharp output correction.

When a corrected error is greater than a certain value, image with acertain rate (in this case, 70%) of the timing information PTS isoutput. This can reduce an image skip.

The following description will describe the case that a process as themaster recovers from its stop.

FIG. 13 is a timing chart showing a processing flow in which a processcorresponding to the original master serves as the master again when theprocess corresponding to the original master is performed again, afterother processing serves as the master in the case that the processcorresponding to the original master stops.

In an illustrated example, in the case that an audio stream stops duringoperation of an audio process as the master, a video process serves asthe master. While, the audio process serves as the master again afterthe audio stream recovers.

FIG. 14 is a timing chart showing a processing flow, in which correctionis continued based on a process as the master by calculating based on aprevious corrected value for a while when a process corresponding to themaster stops, and other processing serves as the master when the processcorresponding to the master does not recover.

In an illustrated example, after a stream stops in the audio 1 process,corrected values corresponding to two times is calculated, and then thecorrection is performed. However, a process as the master does notrecover, thus, the audio 2 process serves as the master. When a processas the master recovers within predetermined time, the process remains asthe master.

FIG. 15 is a timing chart showing a processing flow, in which, when astream process corresponding to the master stops, in consideration of aprevious error of corrected value, when the previous error of correctedvalue is grater than a reference value, other process is changed as themaster, while when the error is less than the reference value, thestream process corresponding to the master remain as the master withoutcorrection for a while.

In an illustrated example, a threshold value is set as “10” to determinethat an error of a previous corrected value is large. In this example,the value is smaller than the threshold value, thus, a process as themaster remains as the master and keep processing for a while. However,the stream for the process as the master does not recover for a while,thus, other process serves as the master. When a process as the masterrecovers within predetermined time, the process remains as the master.

FIG. 16 is a timing chart showing a processing flow, in which correctionis omitted when an average error of corrected values in several times ofprevious correction, so as to reduce overhead, in the case thatcorrection is performed based on a process as the master. Dotted-lineupward arrows represent omission of the correction.

In an illustrated example, the correction is omitted when an averageerror in previous two times is not more than 3. This can reduceoverhead.

FIG. 17 is a timing chart showing a processing flow, in which referenceis omitted when an error between reference values in the latest severaltimes and PTS of a stream is small, in the case that each device otherthan the master references a clock. Dotted-line downward arrowsrepresent omission of the reference.

In an illustrated example, an average of errors between the referencevalues and PTS in first time, and second time is small. Accordingly,next reference is omitted, and the text process provides the text withthe same timing information PTS based on calculation of a clock value.This can reduce overhead of reference.

An AV synchronization system according to the present inventiondigital-decompresses a plurality of streams such as an audio stream, avideo stream, and corrects a system clock by a stream process as themaster. The AV synchronization system can preferably maintainsynchronization even when a stream corresponding to a process as themaster stops or completed in the case of reproduction, in which streamsare synchronized with each other. In addition, it is advantageous toprevent an image skip, a sound skip, and so on, and to reduce overheadof correction and reference.

The present invention is described with concrete embodiments above.However, the conjunction or arrangement of the components according tothe embodiments of the present invention may be embodied in otherspecific forms without departing from the spirit or the appended claims.

1. An AV decoding system comprising: a multi-separating portion forseparating a multiplexed stream, in which a plurality of streams aremultiplexed, into each of the streams including a first stream and asecond stream; a decoding portion for decoding the streams including thefirst stream and the second stream, a data providing portion forproviding decoded data from the decoding portion with synchronized witha system clock; and a time information managing portion for acquiringtime information from the decoded streams, and correcting the systemclock based upon the acquired time information, wherein the timeinformation managing portion switches the decoded first stream to thedecoded second stream as the decoded stream from which the timeinformation managing portion acquires time information.
 2. The AVdecoding system according to claim 1, wherein when the time informationmanaging portion fail to acquire time information from the decoded firststream, the time information managing portion switches the decoded firststream to the decoded second stream as the decoded stream from which thetime information managing portion acquires time information.
 3. The AVdecoding system according to claim 1, wherein when the decoded firststream stops, the time information managing portion switches the decodedfirst stream to the decoded second stream as the decoded stream fromwhich the time information managing portion acquires time information.4. The AV decoding system according to claim 1, wherein when the decodedfirst stream is exhausted, the time information managing portionswitches the decoded first stream to the decoded second stream as thedecoded stream from which the time information managing portion acquirestime information.
 5. The AV decoding system according to claim 1,wherein when the decoded first stream is broken, the time informationmanaging portion switches the decoded first stream to the decoded secondstream as the decoded stream from which the time information managingportion acquires time information.
 6. The AV decoding system accordingto claim 1, wherein when the first stream included in the multiplexedstream inputted to the a multi-separating portion is exhausted, the timeinformation managing portion switches the decoded first stream to thedecoded second stream as the decoded stream from which the timeinformation managing portion acquires time information.
 7. The AVdecoding system according to claim 1, wherein the time informationmanaging portion switches the decoded stream from which the timeinformation managing portion acquires time information among the decodedstreams.
 8. An AV decoding system comprising: a multi-separating portionfor separating a multiplexed stream, in which a plurality of streams aremultiplexed, into each of the streams including a first stream and asecond stream; a decoding portion for decoding the streams including thefirst stream and the second stream, a data providing portion forproviding decoded data from the decoding portion with synchronized witha system clock; and a time information managing portion for acquiringtime information from the decoded streams, and correcting the systemclock based upon the acquired time information, wherein the timeinformation managing portion switches the decoded first stream to thedecoded second stream as the decoded stream from which the timeinformation managing portion acquires time information, wherein the timeinformation managing portion switches the decoded stream from which thetime information managing portion acquires time information among thedecoded streams, and wherein when the time information managing portionfail to acquire time information from the decoded first stream, the timeinformation managing portion switches the decoded first stream to thedecoded second stream as the decoded stream from which the timeinformation managing portion acquires time information, and after thetime information managing portion switches the decoded first stream intothe decoded second stream as the decoded stream from which the timeinformation managing portion acquires time information as a result offailing to acquire time information from the decoded first stream, whentime information from the decoded first stream becomes available to thetime information managing portion, the time information managing portionswitches the decoded second stream to the decoded first stream as thedecoded stream from which the time information managing portion acquirestime information.
 9. An AV decoding system comprising: amulti-separating portion for separating a multiplexed stream, in which aplurality of streams are multiplexed, into each of the streams includinga first stream and a second stream; a decoding portion for decoding thestreams including the first stream and the second stream, a dataproviding portion for providing decoded data from the decoding portionwith synchronized with a system clock; and a time information managingportion for acquiring time information from the decoded streams, andcorrecting the system clock based upon the acquired time information,wherein the time information managing portion switches the decoded firststream to the decoded second stream as the decoded stream from which thetime information managing portion acquires time information, wherein thetime information managing portion switches the decoded stream from whichthe time information managing portion acquires time information amongthe decoded streams, and wherein when the decoded first stream stops,the time information managing portion switches the decoded first streamto the decoded second stream as the decoded stream from which the timeinformation managing portion acquires time information, and after thetime information managing portion switches the decoded first stream tothe decoded second stream as the decoded stream from which the timeinformation managing portion acquires time information as a result ofstop of the decoded first stream, when the decoded first streamrecovers, the time information managing portion switches the decodedsecond stream to the decoded first stream as the decoded stream fromwhich the time information managing portion acquires time information.10. An AV decoding system comprising: a multi-separating portion forseparating a multiplexed stream, in which a plurality of streams aremultiplexed, into each of the streams including a first stream and asecond stream; a decoding portion for decoding the streams including thefirst stream and the second stream, a data providing portion forproviding decoded data from the decoding portion with synchronized witha system clock; and a time information managing portion for acquiringtime information from the decoded streams, and correcting the systemclock based upon the acquired time information, wherein the timeinformation managing portion switches the decoded first stream to thedecoded second stream as the decoded stream from which the timeinformation managing portion acquires time information, wherein the timeinformation managing portion switches the decoded stream from which thetime information managing portion acquires time information among thedecoded streams, and wherein when the decoded first stream is exhausted,the time information managing portion switches the decoded first streamto the decoded second stream as the decoded stream from which the timeinformation managing portion acquires time information, and after thetime information managing portion switches the decoded first stream tothe decoded second stream as the decoded stream from which the timeinformation managing portion acquires time information as a result ofexhaustion of the first decoded stream, when the decoded first streamrecovers, the time information managing portion switches the decodedsecond stream to the decoded first stream as the decoded stream fromwhich the time information managing portion acquires time information.11. An AV decoding system comprising: a multi-separating portion forseparating a multiplexed stream, in which a plurality of streams aremultiplexed, into each of the streams including a first stream and asecond stream; a decoding portion for decoding the streams including thefirst stream and the second stream, a data providing portion forproviding decoded data from the decoding portion with synchronized witha system clock; and a time information managing portion for acquiringtime information from the decoded streams, and correcting the systemclock based upon the acquired time information, wherein the timeinformation managing portion switches the decoded first stream to thedecoded second stream as the decoded stream from which the timeinformation managing portion acquires time information, wherein the timeinformation managing portion switches the decoded stream from which thetime information managing portion acquires time information among thedecoded streams, and wherein when the decoded first stream is broken,the time information managing portion switches the decoded first streamto the decoded second stream as the decoded stream from which the timeinformation managing portion acquires time information, and after thetime information managing portion switches the decoded first stream tothe decoded second stream as the decoded stream from which the timeinformation managing portion acquires time information as a result ofbreak of the decoded first stream, when the decoded first streamrecovers, the time information managing portion switches the decodedsecond stream to the decoded first stream as the decoded stream fromwhich the time information managing portion acquires time information.12. An AV decoding system comprising: a multi-separating portion forseparating a multiplexed stream, in which a plurality of streams aremultiplexed, into each of the streams including a first stream and asecond stream; a decoding portion for decoding the streams including thefirst stream and the second stream, a data providing portion forproviding decoded data from the decoding portion with synchronized witha system clock; and a time information managing portion for acquiringtime information from the decoded streams, and correcting the systemclock based upon the acquired time information, wherein the timeinformation managing portion switches the decoded first stream to thedecoded second stream as the decoded stream from which the timeinformation managing portion acquires time information, wherein the timeinformation managing portion switches the decoded stream from which thetime information managing portion acquires time information among thedecoded streams, and wherein when the first stream included in themultiplexed stream inputted to the a multi-separating portion isexhausted, the time information managing portion switches the decodedfirst stream to the decoded second stream as the decoded stream fromwhich the time information managing portion acquires time information,and after the time information managing portion switches the decodedfirst stream to the decoded second stream as the decoded stream fromwhich the time information managing portion acquires time information asa result of exhaustion of the first stream, when the first streamrecovers, the time information managing portion switches the decodedsecond stream to the decoded first stream as the decoded stream fromwhich the time information managing portion acquires time information.13. An AV decoding system comprising: a multi-separating portion forseparating a multiplexed stream, in which a plurality of streams aremultiplexed, into each of the streams including a first stream and asecond stream; a decoding portion for decoding the streams including thefirst stream and the second stream, a data providing portion forproviding decoded data from the decoding portion with synchronized witha system clock; and a time information managing portion for acquiringtime information from the decoded streams, and correcting the systemclock based upon the acquired time information, wherein the timeinformation managing portion switches the decoded first stream to thedecoded second stream as the decoded stream from which the timeinformation managing portion acquires time information, and wherein thefirst stream is an audio stream and the second stream is a video stream.14. An AV decoding system comprising: a multi-separating portion forseparating a multiplexed stream, in which a plurality of streams aremultiplexed, into each of the streams including a first stream and asecond stream; a decoding portion for decoding the streams including thefirst stream and the second stream, a data providing portion forproviding decoded data from the decoding portion with synchronized witha system clock; and a time information managing portion for acquiringtime information from the decoded streams, and correcting the systemclock based upon the acquired time information, wherein the timeinformation managing portion switches the decoded first stream to thedecoded second stream as the decoded stream from which the timeinformation managing portion acquires time information, wherein the timeinformation managing portion switches the decoded stream from which thetime information managing portion acquires time information among thedecoded streams, and wherein the first stream is an audio stream and thesecond stream is a video stream.
 15. An AV decoding system comprising: amultiplexed stream inputting portion for inputting a multiplexed streamincluding an audio stream and a video stream, and a data outputtingportion for outputting audio data produced based on the audio stream andvideo data produced based on the video stream, wherein the outputtingportion outputs the audio data and the video data according to timeinformation included in the audio stream and time information includedin the video stream alternatively, wherein when the time informationincluded in the audio stream is skipped from a first point to a secondpoint, outputted video data from the outputting portion is skipped froma point corresponding to the first point of the time informationincluded in the audio stream to a point corresponding to the secondpoint of the time information included in the audio stream.
 16. An AVdecoding system comprising: a multiplexed stream inputting portion forinputting a multiplexed stream including an audio stream and a videostream, and a data outputting portion for outputting audio data producedbased on the audio stream and video data produced based on the videostream, wherein the outputting portion outputs the audio data and thevideo data according to time information included in the audio streamand time information included in the video stream alternatively, whereinthe outputting portion outputs text data in addition to the audio dataand the video data, and while the outputting portion outputs the textdata according to the time information included in the audio stream,when the time information included in the audio stream is skipped from afirst point to a second point, the text data outputted from theoutputting portion is skipped from a point corresponding to the firstpoint of the time information included in the audio stream to a pointcorresponding to the second point of the time information included inthe audio stream, and while the outputting portion outputs the text dataaccording to the time information included in the video stream, when thetime information included in the video stream is skipped from a thirdpoint to a fourth point, the text data outputted from the outputtingportion is skipped from a point corresponding to the third point of thetime information included in the video stream to a point correspondingto the fourth point of the time information included in the videostream.
 17. An AV decoding system comprising: a multiplexed streaminputting portion for inputting a multiplexed stream including an audiostream and a video stream, and a data outputting portion for outputtingaudio data produced based on the audio stream and video data producedbased on the video stream, wherein the outputting portion outputs theaudio data and the video data according to time information included inthe audio stream and time information included in the video streamalternatively, wherein while the audio stream included in themultiplexed stream exists, the outputting portion outputs the video dataproduced based on the video stream according to time informationincluded in the audio stream, and when the audio stream included in themultiplexed stream is exhausted, the outputting portion outputs thevideo data produced based on the video stream according to timeinformation included in the video stream.
 18. The AV decoding systemaccording to claim 17, wherein while the outputting portion outputs thevideo data produced based on the video stream according to timeinformation included in the video stream as a result of exhaustion ofthe audio stream included in the multiplexed stream, when the audiostream included in the multiplexed stream is restored, the outputtingportion outputs the video data produced based on the video streamaccording to time information included in the audio stream.
 19. An AVdecoding system comprising: a multiplexed stream inputting portion forinputting a multiplexed stream including an audio stream and a videostream, and a data outputting portion for outputting audio data producedbased on the audio stream and video data produced based on the videostream, wherein the outputting portion outputs the audio data and thevideo data according to time information included in the audio streamand time information included in the video stream alternatively, whereinwhile the audio stream included in the multiplexed stream exists, theoutputting portion outputs the video data produced based on the videostream according to time information included in the audio stream, andwhen the audio stream included in the multiplexed stream is stops, theoutputting portion outputs the video data produced based on the videostream according to time information included in the video stream. 20.The AV decoding system according to claim 19, wherein while theoutputting portion outputs the video data produced based on the videostream according to time information included in the video stream as aresult of stop of the audio stream included in the multiplexed stream,when the audio stream included in the multiplexed stream is restored,the outputting portion outputs the video data produced based on thevideo stream according to time information included in the audio stream.21. An AV decoding system comprising: a multiplexed stream inputtingportion for inputting a multiplexed stream including an audio stream anda video stream, and a data outputting portion for outputting audio dataproduced based on the audio stream and video data produced based on thevideo stream, wherein the outputting portion outputs the audio data andthe video data according to time information included in the audiostream and time information included in the video stream alternatively,wherein while the audio stream included in the multiplexed streamexists, the outputting portion outputs the video data produced based onthe video stream according to time information included in the audiostream, and when the audio stream included in the multiplexed stream isbroken, the outputting portion outputs the video data produced based onthe video stream according to time information included in the videostream.
 22. The AV decoding system according to claim 21, wherein whilethe outputting portion outputs the video data produced based on thevideo stream according to time information included in the video streamas a result of break of the audio stream included in the multiplexedstream, when the audio stream included in the multiplexed stream isrestored, the outputting portion outputs the video data produced basedon the video stream according to time information included in the audiostream.
 23. An AV decoding system comprising: a multiplexed streaminputting portion for inputting a multiplexed stream including an audiostream and a video stream, and a data outputting portion for outputtingaudio data produced based on the audio stream and video data producedbased on the video stream, wherein the outputting portion outputs theaudio data and the video data according to time information included inthe audio stream and time information included in the video streamalternatively, wherein while the audio stream included in themultiplexed stream exists, the outputting portion outputs the video dataproduced based on the video stream according to time informationincluded in the audio stream, and when the time information included inaudio stream is unavailable, the outputting portion outputs the videodata produced based on the video stream according to time informationincluded in the video stream.
 24. The AV decoding system according toclaim 23, wherein while the outputting portion outputs the video dataproduced based on the video stream according to time informationincluded in the video stream as a result of unavailability of the audiostream included in the multiplexed stream, when the time informationincluded in audio stream is available, the outputting portion outputsthe video data produced based on the video stream according to timeinformation included in the audio stream.